Repository navigation
feat(fit): berth fit NAME shows where a request would run and why - #323
Merged
Merged
Conversation
berth fit ranks what it takes to bring an engine, or every engine that serves a model, up right now: resume from RAM, a free card, a drain of the holder, or blocked by a VM, a cordon or a pinned lease. Each candidate names its reason, its weights' tier and the gateway's measured switch time from that tier, or unknown. It reads the same fit.Rank the gateway picks through, so its answer is the gateway's. --explain lists every candidate.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Berth Next milestone 2. #322 put the gateway's choice in
internal/fit. This PR lets people and agents ask thesame question from the CLI: if a request came in for this right now, where would it go, and what would the start
cost?
What changes
berth fit NAME [--explain] [--config] [--costs] [--socket] [--format].[[engine]]name, a discovered engine, or a model in some engine'sserves.resolveranks.--explain, or any answer with more than one candidate, adds the ranked table.unreachable.
docs/interfaces.mdlist the command.berth simulate --addstays a later slice, because it needs Fit to pack sizes, not only rank paths.Proof
TestFitAgainstADaemonruns a real in-process berthd with one free card (gpu0). Modelmis served byaongpu1, which berthd does not report, and by
bon gpu0.b, free, even thoughacomes first in berth.toml.acomes second, blocked, so Fit ranks bypath before config order.
berth fit comfyuiprintscomfyui: blocked, sft-4b-3 holds the cardwith start unknown.berth fit router-strata --explainresolves the model tostrata-sc117, which matches the gateway's pick.proof.shfull passes.